Assessing Machine Translation Quality with Error Analysis

Translation quality can be evaluated with regard to different aspects, such as accuracy (fidelity), fluency and fitness for purpose. In using a machine translation system for information purposes, accuracy of semantic content is the key aspect of quality. Automated quality metrics developed in the machine translation field have been criticized for conflating fluency of form with accuracy of content and for failing to provide any information on the types of errors in the translations. Our research aims to discover criteria for assessing translation quality specifically in terms of accuracy of semantic content in translation. This paper demonstrates how an error analysis with a view to identifying different error types in machine translations can serve as a starting point for such criteria. The error classification described focuses on mismatches of semantic components (individual concepts and relations between them) in the source and target texts. We present error analysis results, which show differing patterns both between human translators and machine translation systems on the one hand and two different kinds of translation systems on the other.